Fix llama-server and llama-bench #11
Conversation
…e model * Add OpenVINO ADD operator to Llama.cpp. The output is somewhat abnormal and needs further debugging.
…ontend-utils, GraphIterator, Decoder
…on openvino device
…ize, iSWA model not working
wine99
force-pushed
the
reset_variable_state
branch
from
abf2454 to
94934fa
Compare
wine99
force-pushed
the
reset_variable_state
branch
from
94934fa to
8f93fe6
Compare
wine99
force-pushed
the
reset_variable_state
branch
from
8f93fe6 to
5f25e52
Compare
cavusmustafa
left a comment
cavusmustafa
left a comment
There was a problem hiding this comment.
I wonder if there is another way to identify different sequences rather than reading kv_cache state. If there is a way to identify, how about creating a unique inference request for each sequence? So, instead of caching inference requests with cgraph pointer, we can create an id out of cgraph+seq_id ? If this is possible, we don't need to read the states as every inference request will manage its own variables anyways.
| break; | ||
| } | ||
| } | ||
| static std::string device = getenv("GGML_OPENVINO_DEVICE") ? getenv("GGML_OPENVINO_DEVICE") : "CPU"; |
There was a problem hiding this comment.
How about doing something like this to avoid multiple getenv calls?
const std::string& getDevice() {
static const std::string device_str= [] {
const char* device_env = std::getenv("GGML_OPENVINO_DEVICE");
return device_env ? std::string(device_env) : "CPU";
}();
return device_str;
}
.
.
.
static std::string device = getDevice();
| ov::AnyMap config; | ||
| if (device == "GPU") { | ||
| auto * disable_sdpa_optimization = getenv("GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION"); | ||
| if (disable_sdpa_optimization && std::string(disable_sdpa_optimization) != "0") { |
There was a problem hiding this comment.
Why do we need to check this for every iteration instead of checking once before compile_model?
|
|
||
| // outdated if: | ||
| // 1. kv_len != kv_len_in_state | ||
| // 2. last row has different values |
There was a problem hiding this comment.
In this case are we deleting the previous kv_cache completely?
wine99
force-pushed
the
dev_backend_openvino
branch
from
956dbf7 to
d5038aa
Compare
llama-server runs with default params (does not support
-np >1)llama-bench runs with
-fa 1